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Abstract —Forensic Examiners within FBI’s Forensic Audio, 
Video and Image Analysis Unit (FAVIAU) perform person 
identification from image and video examinations, such as facial 
comparison, ear comparison, hand comparison, and height 
determination. The examiners currently perform these tasks 
without the assistance of automated biometric applications. Due 
to the high volume of casework that the FAVIAU handles, it is 
anticipated that automated biometrics can be added to the 
examination workflow to increase efficiency, as well as provide 
statistics and rates of probability that can be used in court. This 
paper will outline the current methods of facial and body 
comparisons, as well as height determinations, in order to 
demonstrate to those developing biometric systems the methods 
and needs of the FBI. 

I, Introduction 

HE Federal Bureau of Investigation (FBI) uses a range 
of biometrics and anthropometries and the distribution of 
these services within the FBI is equally widespread. 
Fingerprint biometric tasks are performed by the Laboratory 
Division and the Criminal Justice Information Services 
(CJIS). DNA analyses are performed within the Laboratory 
Division. Additional biometric tasks, such as voice 
comparison and identification of people from images, are the 
responsibility of the Forensic Audio, Video and Image 
Analysis Unit (FAVIAU), within the Operational T echnology 
Division (OTD). This paper will focus on visual biometrics 
applied to the image and video evidence submitted to the 
FAVIAU of the FBI. 

FAVIAU image examiners currently perform facial and 
body comparisons as one-to-one or one-to-few (generally less 
than 10) verifications, as well as performing height 
determinations. These identification tasks are done as part of 
investigations in both criminal and intelligence cases. Often 
these cases require that FAVIAU image examiners testify in 
the judicial system as e?q3ert witnesses. Part of any successful 
testimony is the ability to explain the methods that were used 
to perform the identification. Just as DNA examiners can 
point to repeating base pair matches to justify an 
identification, image examiners must be able to point to 
actual physical features on a face or body to justify tiieir 
conclusions in court. The acceptance of scientific testimony 
and evidence in Federal court is regulated by several 
important legal decisions: Daubert v. MerreU Dow 
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Pharmaceuticals, Inc., 113 S.Ct. 2786 (1993), General 
Electric v. Joiner, 522 U.S. 136 (1997) and Kumho Tire Co. 
v. Carmichael, 526 U.S. 137 (1999); and Federal Rule of 
Evidence 702. Factors considered for acceptance in the court 
include the overall acceptance of the theory and/or technique 
in tile scientific community, the reliability and testability of 
theories and/or techniques, the existence of controls and 
standards, and the existence of an error rate of the theory 
and/or technique. Therefore, the development and evaluation 
of the statistics of personal individualization and of biomettic 
systems is of prime importance to FAVIAU. Likewise, there 
is a critical need to identify the specific physical 
characteristics used by biometric systems to rank candidate 
matches. 

Widiin the FBI, CJIS is undertaking the Next Generation 
Identification Program (NGI), which will e?^and the 
capabilities of the Integrated Automated Fingerprint 
Identification System (lAFIS) to include facial and other 
biometrics [1]. The image examiners of the FAVIAU will be 
the subject matter e:^erts (SME) performing the final facial 
comparisons, or verifications. Therefore, the goal of this 
paper is to demonstrate the processes, and observable 
characteristics, used by the human SME to perform facial and 
body comparisons in order to 1) provide insight to those 
developing biometric systems about how these tasks have 
historically been performed, 2) illustrate what is needed for 
the results of a biometric system to be accepted in the judicial 
system and 3) stress the importance of gathering statistical 
data about the physical characteristics of the human body and 
the reliability of results generated by biometric systems. 

II. Examinations 

There are two types of examinations that involve 
biometrics performed by image examiners within FAVIAU: 
photographic comparisons and photogrammetry (height 
determination). In both of these examinations, images 
depicting tiie subject, tiie ‘q’^^stioned individual’, are 
submitted to the FBI for analyses. The source of the images 
may be a variety of evidential media ranging from 
surveillance video from a bank to film still images recovered 
from a suspect’s camera to digital still images found on a 
questioned individual’s computer. Currently, in both types of 
cases, the examinations are performed without the assistance 
of any type of automated biomefric technology. 

A. Photographic Comparisons 

Photographic comparisons of people are generally 
one-to-one or one-to few comparisons of persons questioned 
and known The overall goal is to determine if the questioned 
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individual is the suspect, the ‘known individual’, to the 
exclusion of all others. If so, this is called Individualization or 
Identification in the forensics community. Note: this would 
be a called a “verification” by the biometrics community. 
Additionally, the examination may lead to the Exclusion or 
Elimination of the known individual as being the questioned 
individual. If an individualization or elimination cannot be 
made, it is reported that no definitive conclusion is possible. 
In every case, even when no conclusion is possible, it is 
common practice to report those features found to be similar 
and dissimilar diat provide the basis for identification or 
exclusion. 

The traits that are used within the photographic comparison 
fall into two categories: class and individual characteristics. 
Class characteristics are those that place an individual wilJiin 
a class or group [2]. These general characteristics include hair 
color, overall facial shape, presence of facial hair, diape of 
the nose, presence of freckles, etc. Individual characteristics 
are those that are unique to the individual and/or allow for a 
person to be individualized [2]. These specific characteristics 
include number and location of facial minutiae, such as 
freckles and blemishes, as well as scars, tattoos, chipped 
teeth, lip creases, wrinkles, etc. By using these traits, 
effectively the ‘texture’ of the face, examiners have been able 
to differentiate between identical twins in images. 
Measurements of facial features and facial landmarks [3,4] 
are compared relatively but not absolutely. Methods that rely 
solely on measurements of facial features or landmarks 
[5,6,7] are severely affected by image perspective and are 
therefore nearly useless in surveillance situations where the 
position of the head is uncontrolled. 

The method of comparison used within FAVIAU follows 
that of fingerprint examiners, the ACE-V method explored in 
[2]. This stands for Analyze, Compare, Evaluate, and Verify. 
In the analysis stage, first the traits of the questioned subject 
are assessed, including class and individual characteristics. 
Second, the traits of the known subject are assessed, including 
class and individual characteristics. Then a comparison of 
these traits and the images is performed. Similarities and 
dissimilarities are noted. If there are dissimilarities, the 
examiner works to understand the nature of the dissimilarity. 
Often dissimilarities can be readily e>q?lained by differences 
in pose, illumination, ejq^ression, or time between the images. 
Once the comparison is made, the examiner evaluates the 
similarities and dissimilarities and their significance. For 
example, the difference in forehead height between 
individuals in questioned and known images is more 
significant than a possible difference in skin-tone as die latter 
can potentially be explained by make-up or tanning. 

The image examiner may or may not enhance the images to 
facilitate the comparison. If the perspectives of the questioned 
and known images are similar, the image examiner can also 
scale the image of the known individual to fiiat of the 
questioned individual by using the interpupilary distance or 
other fixed features within the image. An overlay of die 
scaled known image and the questioned image can be made in 


order to determine if the relative alignment of other facial 
features is consistent. The image examiner can then work 
back and forth between questioned and known imagery to 
assist in their evaluation of similarities and dissimilarities. 
The work is then verified; in FAVIAU this involves a 
peer-review by comparably trained and knowledgeable image 
examiners. 

The conclusions are of 100% certainty in the examiner’s 
opinion fiiat the questioned and known individuals are the 
same individual, are not the same individual, or could not be 
identified or excluded. At this time, percentages or 
probabilities of a match in a facial comparison cannot be 
provided because there is no statistical basis for such 
quantifications. The lack of statistics concerning the 
uniqueness of a face means that no error rate can be provided. 
Therefore, conclusions are reported as an examiner’s expert 
opinion. 

Comparison of ears has proved exceptionally useful to 
image examiners within FAVIAU. Studies have illustrated 
die differentiation between ears of individuals [8,9] and it is 
suggested that each ear is unique. Again, die lack of robust 
statistics is a hindrance to providing a percentage certainty in 
each case. The pronounced 3D nature of ears makes ear 
comparisons exhemely sensitive to image perspective and 
therefore ear comparisons are most effective when the images 
of individuals are of similar perspectives. Another difficulty 
in ear comparisons is obsttuction of the ears, by hair or 
headwear. Nonetheless, FAVIAU examiners have used ears 
to assist in the identification and elimination of subjects of 
comparison. 

Hands, fingers, torsos, arms, and other body parts can be 
compared using similar methods. Hands often feature 
prominently in images of child pornography, as well as 
‘trophy shots’ from homicides and kidnappings. The 
comparison of hands involves noting the position, number, 
and relative pattern of creases on knuckles and palms. The 
presence and morphology of freckles and scars is of 
importance, as well as the pattern and density of hair on the 
back of the hand. Comparisons of arms, legs, torsos, and other 
body parts also generally rely on the morphology, pattern, and 
relative spacing of minutiae and hair. It is not uncommon for 
an. examiner to identify a suspect by linking multiple images 
from either the same crime scene or different ones by 
repeatedly observing similar features. 

The advantages to adding automated facial, ear, and/or 
hand biometrics to this process are clear: increased efficiency, 
as well as a numerical probability of a match (i.e., an error 
rate) that can be illustrated in court. The results provided by 
biometric software would be beneficial by reducing the 
number of suspects and also providing the human examiner 
wifti additional evidence to support their opinion. Currently, 
there is difficulty in that many biomettic systems do not 
provide clear justifications for their numerical results, which 
could cause confusion in a court of law. For example, one 
biometric system may yield results showing that two identical 
images of a person have a match score of 10 while two 




different images of the same person may have a match score 
of 30; these discrepancies must be readily explainable for die 
technology to be used in the judicial system. A statistical 
basis for the conclusions would be ideal but at a minimum die 
results should be explicable and consistent. It is not necessary 
for a biometric system to illustrate its proprietary algoridim, 
however it would be beneficial if the reasoning for a match 
from a specific system had an explainable basis in observable 
traits. 

B. Photogrammetry 

The objective of photogrammetry is the indirect 
determination of spatial measurements of objects using 
photographic images [10]. In FAVIAU’s forensic context, 
photogrammetry is most often used to determine the height of 
a questioned individual or to determine the length of a 
weapon used in the commission of a crime. The 
determination of a questioned individuaTs height is generally 
done by one of two methods: reverse projection or analytical 
photogrammetry. This work will not go into the details of 
either process. Instead, it is enough to note that reverse 
projection photogrammetry involves Image Examiners 
reconstructing the original scene of the image and then taking 
an additional image with a known scale, e.g. a height chart, 
placed into the scene. Analytical photogrammetry at the FBI 
involves using geometry and perspective analysis to establish 
the proportions of the scene and thus determine the height of a 
questioned individual within that scene. 

Height determination is commonly performed to eliminate 
suspects or to link multiple crimes, but never as a means to 
individualize a subject. It is anticipated that gait analysis 
could be used in conjunction with photogrammetry to assist in 
the identification of a questioned individual. Also, 
determined heights would allow an Examiner to focus on a 
narrower range of individuals when querying biometric 
databases. 

III. Research Interests 

FAVIAU has a strong interest in developing biomehic 
systems to assist forensic investigations, increasing efficiency 
and productivity. Additionally, the statistics of personal 
identification are of interest to FAVIAU as such data serve to 
establish the uniqueness of individuals. For example, it would 
be beneficial to know what percentage of the population has a 
certain ear length or specific pattern of knuckle creases. 
These statistics are surprisingly lacking, even though they 
could provide the numerical certainty that would allow 
biometrics, including fingerprints, smoother passage tiirough 
the judicial system. Consider how the statistics provided by 
DNA analysis add to its appeal in the legal system. Therefore, 
FAVIAU has funded biomettic research projects that involve 
both the development of biometric systems and increasing tiie 
known statistics of human identification. A recent project at 
the University of Sheffield in the UK (the ‘‘Magna database”) 
involved using 2D and 3D images of faces for discriminating 
the nature of the available cranio-facial landmarks, and 


derivative measurements, to build statistical models of facial 
measurements within the sample population [11]. Another 
project, undertaken at West Virginia University and 
University of Miami, involved using images of ears to 
develop 3D ear models and then discriminating between ears 
[12]. 

FAVIAU anticipates building on both the facial and ear 
research projects that we have previously funded, as well as 
ejq)loring additional biometrics including hands and gait. 
Furtiier facial studies would include additional data collection 
and landmarking, to make the facial measurement statistics 
more robust and quantifiable. Additionally, future facial 
studies will include detailed analyses of facial minutiae, or 
blemishes, such as moles, scars, and freckles. The frequency 
of facial minutiae has not been assessed but needs to be 
determined as such statistics would significantly improve our 
ability to individualize persons to a high probability. 

Likewise, additional ear analyses are needed. Instead of 
databases of hundreds of ears, thousands of ears, or more, 
need to be captured, studied, tested, and verified. 
Uncontrolled views of ears, including obstructed views due to 
objects such as hair or headwear, must also be considered. 
The ability to develop a three dimensional model of a known 
ear would allow an examiner to match the orientation of a 
questioned ear observed in an uncontrolled video, a common 
scenario in forensic and intelligence cases, thus allowing for 
both automated and manual comparisons to be made. 

Research studies focusing on die measurements of hands 
and the patterns of knuckle creases are also of interest to the 
FBI. The development of measurements, and thus statistics, 
to quantify the uniqueness of hands and determination of the 
potential uniqueness of knuckle creases are desired. 

Additionally, gait analysis is another research avenue of 
interest to FAVIAU. Video from scenes such as bank 
robberies and surveillance situations are sometimes captured 
at frames rates considered to be real time, or close to real 
time. These higher frame rates depict nearly continuous 
motion of a subject that is walking or running. Images that 
depict an entire person who is in motion are often unsuitable 
for facial or ear photographic comparison or biometrics and 
are also challenges for photogrammetric height analysis, as 
the stride of the person affects die ability to measure their true 
stature. Therefore, the ability to use gait analyses to 
individualize persons in motion would greatly increase our 
forensic capabilities. Combining gait analysis and 
photogrammehic height analysis would further assist in the 
individualization of questioned persons. 

Lastiy, the multi-modal combination of biometrics 
including face and ear, hand measurements and knuckle 
patterns, and gait and height is an ultimate goal of the 
biometrics program at the FBI. 

IV. Summary 

The use of automated biometric systems aimed at facial 
recognition, ear identification, hand identification, gait 



analysis, and height determination would greatly enhance die 
efficiency of forensic work performed by the FBI. However, 
to date, no biometric system performing the aforementioned 
tasks has been accepted within the judicial system. Biometric 
systems must continuously be evaluated and verified to be 
accepted within a court of law. Likewise, the results should 
be explainable. Additionally, the statistics of human 
identification are of prime importance to the forensic 
community. 
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